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13 December 1976 


MEMORANDUM 


SUBJECT: Report on the Origin, Procedures, and Status of the 
Experiment in Competitive Analysis on National 
Intelligence Issues 


1. This memorandum reports on the experiment in competitive 
analysis undertaken in parallel with the preparation of the 
1976 National Intelligence Estimate on Soviet Forces for Inter- 
continental Conflict (NIE 11-3/8-76). In this report I will cover 
points on the origin of the experiment; its purposes and objectives; 
the institutional framework within which the agreement to conduct 
the study was reached; procedures for implementing it; progress _ 
to date; and a tentative and preliminary evaluation of the steps 
so far completed. Documentary attachments are provided as 
appropriate. 


2. The request to conduct an exercise in competitive 
analysis was initially raised by the President's Foreign Intelligence 
Advisory Board (PFIAB) with the President in August 1975. The Board 
felt that the 1974 NIE on Soviet Forces for Intercontinental Conflict 
was seriously misleading in the presentation of a number of key 
judgments and that it projected a sense of complacency in regard 
to the Soviet threat unsupported by the facts. Subsequently 
Dr. Kissinger asked the DCI -- Mr. Colby -- to provide the 
President with comments on the experiment proposed by the PFIAB. 
In his reply, Mr. Colby suggested that the forthcoming NIE on 
Soviet strategic forces be reviewed by the Advisory Board to 
determine the extent to which the new estimate overcame deficiencies 
perceived in the 1974 estimate. This, in Mr. Colby's view, would 
provide a better basis for determining if the experiment should 
be purused. Mr. Colby's suggestion was adopted and an Estimates 
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Evaluation Committee to conduct the review was established 
within the PFIAB. The Committee consisted of Mr. Robert Galvin 
as Chairman, with Dr. John S. Foster and Dr. Edward Teller 

as members. 


3. In April 1976, the Evaluation Committee reported to 
the Board on their review of the 1975 Estimate. They concluded 
that while some improvements had been noted it was still, in 
their view, worthwhile to pursue the original proposal to 
establish an experimental competitive analysis group. The 
Committee reached its conclusions after (a) comparing the 1974 
and 1975 estimates: (b) reviewing a ten-year "track record" 
of NIE judgments about Soviet strategic programs; and (c) 
conducting a survey of some 40 "authorities" in and outside 
of government. The Board Supported the Committee recommendation. 
Subsequently the DCI met with PFIAB to discuss the findings of 
the Estimates Evaluation Committee. (Attachment A) . 

~ 4, In early May of 1976 the Deputy to the DCI for National 
Intelligence, at that time Mr. George Carver, met with Mr. Robert 
Galvin to discuss in greater detail the methods, procedures, and 
objectives for conducting the experiment. The agreement finally 
worked out and coordinated with General Scowcroft -- the 
Assistant to the President for National Security Affairs -- is 
provided in a letter from the Chairman of PFIAB to Mr. Colby's 
Successor, Mr. George Bush, dated 8 June 1976 (Attachment B). 
The principal provisions were as follows: 


The 1976 estimate of Soviet forces for intercontinental | 
attack would be prepared in accordance with established 
Community practices. Those working on the estimate would 
be referred to as the "A" team. 


In addition, with respect to three key issues selected 
by the DCI in consultation with General Scowcroft, "B" 
teams would be formed of experts inside or outside of 
government. These persons should have expert knowledge 
in the appropriate fields, who have or can be granted 
the necessary security Clearances, but who are not 
themselves engaged in the production of the basic 
Community estimate. 


ra 
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The "A" and "B" teams are to have the same body of 
information available to them and will adhere to the same 
| production schedule. When the studies are completed the 
teams will have an opportunity to interact on their 
findings -- not to strike compromises or reach consensus 
judgments but to explain and defend their findings before 
their peers. 


Final "A" and "B" drafts will then be produced, and each 
team will be able to prepare written comments on the other's 
findings. The entire package will be considered by the 
NFIB. The Intelligence Community estimate will -be 

| handled in standard ways. The "B" team studies and the 

"A" and "B" team comments on each other's work will be 
packaged separately and forwarded only to selected 
recipients -- among whom will be the PFIAB and the Special 
Assistant to the President for National Security Affairs. 


Later, the Assistant to the President for National 

Security Affairs, in consultation with the DCI and 
PFIAB, will review the experiment and critique its 

results. 


5. Mr. Bush designated el ~~ formerly Deputy 
Director of CIA's Office of Strategic Research -- as the 
Intell igence_Communi rdinator and manager for this 
Seren eee ee] role was to provide a central focus 
and support for the “B" teams -- with a responsibility to 
follow substantive progress but not to intervene in the analysis 
of the competitive teams. The three topics for study were selected 
in consultation with the NSC Staff and the Estimates Evaluation 
Committee of PFIAB. These were Soviet ICBM accuracy, Soviet 
low altitude air defense capability, and Soviet Strategic policy 
and objectives (Attachment C). Guidance on the composition of 
the "B" teams was received from the PFIAB and the teams were 
structured to conform reasonably closely to those guidelines. 
In view of the short notice, many of the persons approached to 
work on the experiment could not do so. However, PFIAB was 
notified of the final composition of each team and Mr. Galvin 
expressed general satisfaction with the progress being made. 
(Attachment D) 
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6. From the outset the experiment was designed to test 
the hypothesis -- using the same evidence as that available 
to the Intelligence Community -- that either the range of 
uncertainty around selected Soviet threat parameters (ICBM 
accuracy, low altitude air defense capabilities) was such that 
the threat could be greater than that reflected in the National 
Estimates or -- in the case of the estimate of Soviet strategic 
objectives -- that the Soviets were pursuing amore comprehensive 
and systematic program for a dominant military capability than 
they were being credited with in the NIEs. The teams were 
advised by the Chairman of the Estimates Evaluation Committee of 
PFIAB, Mr. Galvin, that they were not to feel obliged to fit 
the evidence to a pre-determined conclusion if, in fact, their 
analyses indicated otherwise. The teams were -- nonetheless -- 
made up of experienced analysts of Soviet military matters whose 
views in the past have coincided with the hypothesis being examined. 
The experiment consequently was not really balanced, in that it 
did not seek out "C" teams who would sympathetically examine 
alternative interpretations of the evidence that might lead to 
more benign evaluations of Soviet military strength and purposes 
than either the Community or the "B" team studies. 


7. The work of the "B" teams was pursued throughout the 
summer and early fall on essentially the schedule establ ished 
in June. The "A" and "B" team drafts were completed and 
exchanged in October and the three sets of teams met on November 3rd 
through the 5th to discuss and defend their positions. Following 
those meetings the "B" teams prepared their final studies. These 
studies are now in process of being prepared for printing. 


8. The existence and nature of the competitive analysis 
experiment were relatively widely known in Washington by mid- 
summer -- but in a classified context -- and all the participants 
had been cautioned about the damage that could be done by 
unauthorized press leaks. As you know, however, the Boston Globe 
and the Washington Star on October 20 published identical articles 
which provided the outline and some of the essential detail of 
the exercise (Attachment E). We have been unable to discover 
the source of this story. Strong representations were made by 
Mr. Bush to all the "A" and "B" team participants concerning 
the negative effect such publicity could have on the utility of 
the experiment. Mr. Bush also conferred with Dr. Cherne, Chairman 
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Of PFIAB, and asked that the Board members also be advised of 

| his views concerning the disutility of further exposure of the 
experiment in the public domain. The individual Board members 
were requested to use any influence they might have with the 
participants in the exercise to prevent further leaks to the 
press. Subsequently there have been no further leaks, although 
there have been a few brief notes in various parts of the press 
that appear to be based only on the original Beecher article. 


9. It was originally anticipated that the basic NIE 
and the "B" team studies would be considered by the NFIB in 
November and the results briefed to the PFIAB at their December 
meeting -- the last scheduled before the inauguration. Although 
the NFIB schedule for NIE 11-3/8-76 slipped into December, the 
PFIAB still wanted to hear from both the Community teams and the 
"B" teams on the results of the experiment completed so far. 
The PFIAB felt that the exercise was close enough to being 
completed that they should report on it to the President at 
their last meeting with him. 


10. The briefing of PFIAB took place on 2 December. The 
intent was not to present final views because the PFIAB 
Chairman, Dr. Cherne, did not wish to have the Board sit in 
judgment on the experiment until al] stages had been completed. 
I think it fair to say, however, that the Board was pleased 
with the results on both substantive grounds and for what it 
represented in the way of handling dissent in the analytical 
process. My own view is somewhat mixed. We will prepare a 
thorough evaluation of the experiment when it is finally over and 
I will provide you with a copy as soon as possible, but I can 
give you a capsule version of how I come down at this time. 


ll. In brief, while there were some positive aspects to 
the experiment I now conclude that they are at least balanced 
by the costs and problems associated with it. On the positive 
side: 


-- In the broadest sense, the experiment has been 
supplemental to other factors, such as new 
evidence, in influencing the estimate, but 
except for two specifics was not the sole or 

| primary factor. 
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-- The interaction on the issue of Soviet strategic 
policies and objectives resulted in some positive 
presentational adjustments to this year's NIE. 


-- Specifically the "A" and "B" team interaction on 
Soviet low altitude air defense capabilities did 
have a perceptible influence on a significant 
judgment in the estimate concerning uncertainty 
about future Soviet capabilities to limit damage 
to the USSR. 


12. The costs and problems associated with this experiment 
have not been fully assessed as yet, but they will fall into the 
following three broad categories: 


-- Financial costs, which in direct accounting will 

be of the order of There are additional 25X 
i housekeeping functions that have not been directly 
attributed to this experiment. 


-- The costs in terms of time spent by the Community 
in the care and feeding of the "B" team, and the 
effect this has had on the bulk of the estimate 
and its schedule. 


-- The future impact likely to be felt from the 
probably inevitable publicity (unauthorized) 
the experiment will receive. There are already 
signs of renewed press interest. Congressional 
enquiries have also been made, though not yet 
pressed. 


13. I will be available_to discuss this with you in more 
detail if you like -- as will I will also keep . 25) 
you informed of important new devefopments if they arise and 
will provide a copy of our final assessment of the experiment 
when it is completed. 


25 


Richard Lehman 
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